System and method for dynamic in-vehicle virtual reality

ABSTRACT

A method for dynamic virtual reality includes receiving vehicle data. The vehicle data includes a velocity, an orientation, and a location of a vehicle within a first frame of reference. The method also includes receiving user data. The user data includes a velocity, an orientation and a location of a user within a second frame of reference different than the first frame of reference. The method includes generating a view based on the vehicle data, the user data, and a model, the model including one or more components that define the view. The method includes determining a difference between the user data and the vehicle data. The method includes rendering the view to the output device by controlling the output device to update display of the view according to the determined difference relative to the second frame of reference.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/420,536 filed Jan. 31, 2017 and now published as U.S. Patent Pub. No.2017/0139473, which is expressly incorporated herein by reference. U.S.patent application Ser. No. 15/420,536 is a continuation of U.S. patentapplication Ser. No. 14/291,854 filed on May 30, 2014 and now publishedas U.S. Patent Pub. No. 2015/0097861, which is expressly incorporatedherein by reference. Further, U.S. patent application Ser. No.14/291,854 is a continuation of U.S. patent application Ser. No.14/177,841 filed on Feb. 11, 2014 and now published as U.S. Patent Pub.No. 2015/0097860, which claims priority to U.S. Provisional ApplicationSer. No. 61/886,240 filed on Oct. 3, 2013, both of which are alsoexpressly incorporated herein by reference.

BACKGROUND

Truly immersive virtual reality and augmented reality systems createenvironments that submerge the perceptual system of a user incomputer-generated stimuli (e.g., a virtual world and/or a virtualview). Typically, these immersive systems captivate the senses of theuser while blocking out stimuli from the physical world. The virtualworld can be altered as a result of an input from the user and/or aninteraction of the user with the virtual world.

Although, the goal of these systems is to create an immersiveenvironment, the physical world can still be used to provide a contextfor the immersive environment. In particular, it is important toconsider the environment and context of the user. For example, in thecontext of a user in a vehicle, immersive virtual reality and augmentedreality systems can consider information about the user, the vehicle andthe user in relation to the vehicle. This information can be used togenerate an immersive environment that is customized to the user and theuser's environment and can allow the user to perceive the immersiveenvironment comfortably.

BRIEF DESCRIPTION

According to one aspect, a method for virtual reality rendering views toan output device is provided. The method includes receiving vehicledata. The vehicle data includes a velocity, an orientation, and alocation of a vehicle within a first frame of reference. The method alsoincludes receiving user data. The user data includes a velocity, anorientation and a location of a user within a second frame of referencethat is different than the first frame of reference. The method furtherincludes generating a view based on the vehicle data, the user data, anda model, the model including one or more components that define theview. The method includes determining a difference between the user datafrom the second frame of reference and the vehicle data from the firstframe of reference. The method yet further includes rendering the viewto the output device by controlling the output device to update displayof the view according to the determined difference relative to thesecond frame of reference.

According to another aspect, a computing system for rendering views toan output device is provided. The computing system a processoroperatively connected for computer communication to one or more vehiclesystems of a vehicle and the output device. The processor includes avirtual reality data module receives vehicle data from the one or morevehicle systems of the vehicle. The vehicle data includes a velocity, anorientation, and a location of the vehicle within a first frame ofreference. The virtual reality data module receives predictive vehicledata including a velocity, an orientation, and a location of the vehiclewithin the first frame of reference at a future point in time. Thevirtual reality data module also receives user data the from the outputdevice. The user data includes a velocity, an orientation and a locationof a user within a second frame of reference of the vehicle. Theprocessor also includes a dynamic virtual reality module that generatesa view based on the vehicle data, the predictive vehicle data, the userdata, and a model, the model including one or more components thatdefine the view. The processor also includes a rendering module thatdetermines a difference between the user data from the second frame ofreference and the vehicle data from the first frame of reference. Therendering module also renders the view to the output device bycontrolling the output device to update display of the view according tothe determined difference relative to the second frame of reference andthe predictive vehicle data relative to the first frame of reference.

According to another aspect, a non-transitory computer-readable storagemedium storing instructions that, when executed by a vehicle computer,causes the computer to perform the steps of a method. The methodincludes receiving vehicle data. The vehicle data includes a velocity,an orientation, and a location of a vehicle. The method also includesreceiving user data. The user data includes a velocity, an orientationand a location of a user. The method further includes generating a viewbased on the vehicle data, the user data, and a model, the modelincluding one or more components that define the view. The methodincludes determining a difference between the user data and the vehicledata. The method yet further includes rendering the view to the outputdevice by controlling the output device to update display of the viewaccording to the determined difference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of an operating environment for dynamicvirtual reality systems and methods according to an exemplaryembodiment;

FIG. 2 is a schematic view of a vehicle and a vehicle occupantimplementing virtual reality systems and methods according to anexemplary embodiment;

FIG. 3 is a block diagram of the virtual reality engine of FIG. 1according to an exemplary embodiment;

FIG. 4 is a virtual world model diagram including data flow according toan exemplary embodiment;

FIG. 5 is a schematic class diagram of a virtual reality world includingthe virtual world model of FIG. 4 according to an exemplary embodiment;

FIG. 6 is a flow chart diagram of a method for dynamic virtual realityaccording to an exemplary embodiment;

FIG. 7A is an illustrative example of a virtual view according to anexemplary embodiment; and

FIG. 7B is an illustrative example of another virtual view according toan exemplary embodiment.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein.The definitions include various examples and/or forms of components thatfall within the scope of a term and that can be used for implementation.The examples are not intended to be limiting.

A “bus”, as used herein, refers to an interconnected architecture thatis operably connected to other computer components inside a computer orbetween computers. The bus can transfer data between the computercomponents. The bus can a memory bus, a memory controller, a peripheralbus, an external bus, a crossbar switch, and/or a local bus, amongothers. The bus can also be a vehicle bus that interconnects componentsinside a vehicle using protocols such as Controller Area network (CAN),Local Interconnect Network (LIN), among others.

“Computer communication”, as used herein, refers to a communicationbetween two or more computing devices (e.g., computer, personal digitalassistant, cellular telephone, network device) and can be, for example,a network transfer, a file transfer, an applet transfer, an email, ahypertext transfer protocol (HTTP) transfer, and so on. A computercommunication can occur across, for example, a wireless system (e.g.,IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system(e.g., IEEE 802.5), a local area network (LAN), a wide area network(WAN), a point-to-point system, a circuit switching system, a packetswitching system, among others.

A “disk”, as used herein can be, for example, a magnetic disk drive, asolid state disk drive, a floppy disk drive, a tape drive, a Zip drive,a flash memory card, and/or a memory stick. Furthermore, the disk can bea CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CDrewritable drive (CD-RW drive), and/or a digital video ROM drive (DVDROM). The disk can store an operating system that controls or allocatesresources of a computing device.

A “database”, as used herein can refer to table, a set of tables, a setof data stores and/or methods for accessing and/or manipulating thosedata stores.

A “memory”, as used herein can include volatile memory and/ornon-volatile memory. Non-volatile memory can include, for example, ROM(read only memory), PROM (programmable read only memory), EPROM(erasable PROM), and EEPROM (electrically erasable PROM). Volatilememory can include, for example, RAM (random access memory), synchronousRAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double datarate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory canstore an operating system that controls or allocates resources of acomputing device.

A “module”, as used herein, includes, but is not limited to, hardware,firmware, software in execution on a machine, and/or combinations ofeach to perform a function(s) or an action(s), and/or to cause afunction or action from another module, method, and/or system. A modulecan include a software controlled microprocessor, a discrete logiccircuit, an analog circuit, a digital circuit, a programmed logicdevice, a memory device containing executing instructions, and so on.

An “operable connection”, or a connection by which entities are“operably connected”, is one in which signals, physical communications,and/or logical communications can be sent and/or received. An operableconnection can include a physical interface, a data interface and/or anelectrical interface.

A “processor”, as used herein, processes signals and performs generalcomputing and arithmetic functions. Signals processed by the processorcan include digital signals, data signals, computer instructions,processor instructions, messages, a bit, a bit stream, or other meansthat can be received, transmitted and/or detected. Generally, theprocessor can be a variety of various processors including multiplesingle and multicore processors and co-processors and other multiplesingle and multicore processor and co-processor architectures. Theprocessor can include various modules to execute various functions.

A “portable device”, as used herein, is a computing device typicallyhaving a display screen with user input (e.g., touch, keyboard) and aprocessor for computing. Portable devices include, but are not limitedto, handheld devices, mobile devices, smart phones, laptops, tablets ande-readers.

A “vehicle”, as used herein, refers to any moving vehicle that iscapable of carrying one or more human occupants and is powered by anyform of energy. The term “vehicle” includes, but is not limited to:cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats,personal watercraft, and aircraft. In some cases, a motor vehicleincludes one or more engines.

A “vehicle system”, as used herein can include, but are not limited to,any automatic or manual systems that can be used to enhance the vehicle,driving and/or safety. Exemplary vehicle systems include, but are notlimited to: an electronic stability control system, an anti-lock brakesystem, a brake assist system, an automatic brake prefill system, a lowspeed follow system, a cruise control system, a collision warningsystem, a collision mitigation braking system, an auto cruise controlsystem, a lane departure warning system, a blind spot indicator system,a lane keep assist system, a navigation system, a transmission system,brake pedal systems, an electronic power steering system, visual devices(e.g., camera systems, proximity sensor systems), a climate controlsystem, an electronic pretensioning system, among others.

Referring now to the drawings, wherein the showings are for purposes ofillustrating one or more exemplary embodiments and not for purposes oflimiting same, FIG. 1 is a schematic view of an operating environment100 for implementing dynamic virtual reality systems and methodsaccording to an exemplary embodiment. The components of environment 100,as well as the components of other systems, hardware architectures andsoftware architectures discussed herein, can be combined, omitted ororganized into different architectures for various embodiments. Further,the components of the operating environment 100 can be implemented withor associated with a vehicle. For example, FIG. 2 illustrates a vehicle200 implementing dynamic virtual reality systems and methods, which willbe described in further detail herein.

In the illustrated embodiment of FIG. 1, the environment 100 of FIG. 1includes a vehicle computing device (VCD) 102 (e.g., a telematics unit,a head unit, a navigation unit, an infotainment unit, an electroniccontrol unit) with provisions for processing, communicating andinteracting with various components of a vehicle (e.g., the vehicle 200,FIG. 2) and other components of the environment 100. Generally, the VCD102 includes a processor 104, a memory 106, a disk 108, a GlobalPositioning System (GPS) 110 and an input/output (I/O) interface 112,which are each operably connected for computer communication via a bus114 (e.g., a Controller Area Network (CAN) or a Local InterconnectNetwork (LIN) protocol bus) and/or other wired and wirelesstechnologies. The I/O interface 112 provides software and hardware tofacilitate data input and output between the components of the VCD 102and other components, networks and data sources, which will be describedherein. Additionally, as will be discussed in further detail with thesystems and the methods discussed herein, the processor 104 includes avirtual reality (VR) engine 116 suitable for providing a dynamic virtualreality environment to a user (e.g., a vehicle occupant) facilitated bythe components of the environment 100.

The VCD 102 is also operably connected for computer communication (e.g.,via the bus 114 and/or the I/O interface 112) to one or more vehiclesystems 118. Vehicle systems can include, but are not limited to, anyautomatic or manual systems that can be used to enhance the vehicle,driving and/or safety. The vehicle systems 118 include and/or areoperably connected for computer communication to various vehicle sensors120, the vehicle sensors 120 providing and/or sensing informationassociated with the vehicle, the vehicle environment and/or the vehiclesystems 118. The sensors 120, can include, but are not limited to,vehicle state sensors, vehicle system state sensors, proximity sensors,vision sensors, audio sensors, GPS sensors (e.g., GPS 110), and othersensors. Specific vehicle system sensors can include, but are notlimited to, vehicle speed sensors, accelerator pedal sensors, brakesensors, throttle position sensors, wheel sensors, anti-lock brakesensors, camshaft sensors, among others. The sensors 120 are operable tosense a measurement of data associated with the vehicle, the vehicleenvironment, the vehicle systems 118, and/or occupants of the vehicle,and generate a data signal indicating said measurement of data. Thesedata signals can be converted into other data formats (e.g., numerical)and/or used by the vehicle systems 118 and/or the VCD 102 to generateother data metrics and parameters. In particular, the VCD 102 andcomponents thereof can access and/or receive data (e.g., vehicle data,user data, other data (FIG. 3)) from the plurality of vehicle systems118 and/or the sensors 120.

Referring again to FIG. 1, the VCD 102 is also operatively connected forcomputer communication to various networks 122 and virtual reality (VR)devices 124. The network 122 is, for example, a data network, theInternet, a wide area network or a local area network. The network 122serves as a communication medium to various remote devices (e.g., webservers, remote servers, application servers, intermediary servers,client machines, other portable devices (not shown)). The VR devices 124include devices that provide input and/or output to the VCD 102, and inparticular, the VR engine 116, to facilitate and provide a dynamicvirtual reality environment to a user. For example, VR devices 124 caninclude, but are not limited to one or more of the following: virtualreality tracking devices, head-mounted displays, virtual realityclothing, virtual reality input devices, virtual reality glasses, cameratracking systems in the vehicle for monitoring the user, the vehicleand/or the vehicle environment, portable devices, mobile devices, amongothers.

In one example, the VR device 124 is a head mounted display (HMD), thatcan be placed on a user's body (e.g., head) or attached on a helmet orgoggles. For example, referring to FIG. 2, a vehicle occupant 202,positioned in a seat 204 of the vehicle 200, is wearing an HMD 206placed on the head of the vehicle occupant 202. The HMD 206 can provideinformation about the vehicle occupant 202, for example, trackinginformation, input information, motion information, among others to theVR engine 116. The HMD 206 also provides a virtual view generated by theVR engine 116 to the vehicle occupant 202. In another embodiment, thevehicle occupant 202 can be in possession of a portable device 208. Theportable device 208, alone or in combination with the HMD 206, canprovide information about the vehicle occupant 202 and provide a virtualview generated by the VR engine 116 on the portable device 208. Forexample, the portable device 208 can include accelerometers, gyroscopesand/or compasses that can provide tracking information about the vehicleoccupant 202 in possession of the portable device 208. The HMD 206and/or the portable device 208 can also include speakers or headphonesfor audio input and output. The HMD 206 and/or the portable device 208can utilize wireless or wired technology for computer communication withthe VCD 102. Various input/output technologies can be implemented withthe systems and methods described herein. Other configurations and usesof VR devices 124 can also be utilized. For example, the vehicleoccupant 202 can also use a controller (not shown, but can beimplemented as the portable device 208) or can use the portable device208 alone or in conjunction with the HMD 206.

The VR engine 116 of FIG. 1 and a system for dynamic virtual realitywill now be discussed in detail with reference to FIG. 3 and FIG. 1.FIG. 3 illustrates a block diagram of a virtual reality (VR) engine 300(e.g., the VR engine 116) according to an exemplary embodiment. Forsimplicity, not all components of FIG. 1 are shown in FIG. 3. The VRengine 300 includes a virtual reality data module 302, a dynamic virtualreality (VR) module 306 and a rendering module 308. In addition to thefunctionality described above with reference to FIG. 1, theaforementioned modules can access and/or receive vehicle data 318, userdata 320 and other data 322, as well as communicate with vehicle systems314 and VR devices 316. As discussed above, the VR devices 316 provideinput and/or output to the VR engine 300. Specifically, the VR devices316 can provide input and/or output to a user 312 (e.g., a vehicleoccupant 202 (FIG. 2)). For example, as discussed above with FIG. 1, theVR devices can include virtual reality tracking devices, head-mounteddisplays (e.g., the HMD 206), portable devices (e.g., the portabledevice 208), virtual reality clothing, virtual reality input devices,among others.

The virtual reality data module 302 receives vehicle data from the oneor more vehicle systems of a vehicle. The vehicle data includes vehicledynamics data. For example, the virtual reality data module 306 canreceive vehicle data 318, from for example, the vehicle systems 314. Thevehicle data 318 includes vehicle data metrics and parameters derivedfrom the vehicle sensors 120. For example, vehicle data can include, butis not limited to, vehicle location data (e.g., from the GPS 110),vehicle orientation data, vehicle system state data, data related to oneor more vehicle systems and/or components, vehicle environment data(e.g., interior and exterior environment data) among others. The vehicledata 318 can also include navigation data, for example, location data,direction data (e.g., origin, destination, point of interest), amongothers.

The vehicle data 318 can also include vehicle dynamics data thatdescribes the dynamics of the vehicle and the motion of vehicle (e.g.,velocity, direction, acceleration, yaw rate, steering rate, steeringangles). Vehicle dynamics data can include, but is not limited to, realtime data concerning the speed level, the acceleration rate, the yawrate, the steering wheel position, the brake position, the throttleposition, the transmission gear position of the vehicle, drivercommands, dynamic car responses, tire and road forces, among others. Insome embodiments, the vehicle data 318 can be received from remotesources, for example, the network 122 (FIG. 1). In one embodiment, theVR data module 302 can receive predictive vehicle data or can determinepredictive vehicle data based on the vehicle data 318 and/or the vehicledynamics data. For example, vehicle predictive motion data can be basedon pedal positions, vehicle system status/control, current vehiclelocation, vehicle destination information, among others.

The virtual reality data module 306 also receives user data 320. Forexample, the user data 320 can be received from the VR devices 316. Theuser data 320 includes tracking data, interaction data, user input data,from for example, the VR devices 316. For example, the user data can bebased at least in part, on data from vision sensors (e.g., sensors 120,cameras, gesture/motion sensors), tracking systems, the HMD 206, theportable device 208, and other sensors and systems that provide dataabout the user's interaction, position, orientation, location andmotion. For example, the user data 320 can include position, orientationand location information about the user. In another embodiment, the userdata 320 can include position, orientation and location informationabout the user in relation to the vehicle, for example, based in leastin part, on the vehicle data 318 (including the vehicle dynamics data).Thus, in one embodiment, the user data 320 can provide information onthe user's motion and position and how the user's motion and position isaffected by the vehicle dynamics. The user data can also include healthdata about the user, for example, from health monitoring devices (e.g.,portable medical devices worn by the user, biological health monitoringdevices). In some embodiments, the user data can also be received fromother networks 122 and/or the vehicle systems 314.

In another embodiment, the virtual reality data module 302 also receivesother data for facilitating dynamic virtual reality. The other data 322can include can include big data from the vehicle systems 314, the VRdevices 316 and/or other networks 122. For example, other data 322 caninclude environmental data associated with the vehicle (e.g., interior,exterior), road conditions (e.g., bumpy roads, slick roads, trafficconditions), weather conditions, vehicle temperature, among others. Inanother embodiment, the other data 322 can include driver action data,for example, driving history, fuel efficiency, interactions with othervehicle systems, gestures, motion relative to the vehicle, among others.Further, in some embodiments, the other data 322 can include socialmedia data from, for example, the other networks 122.

Referring again to FIG. 3, the dynamic VR data module 306 generates avirtual view based on the vehicle data, the user data and a virtualworld model. In one embodiment, a data store stores a virtual worldmodel, the virtual world model including one or more components thatdefine the virtual view. For example, in FIG. 3, a data store 310 canstore a virtual world model and design data. The virtual world model anddesign data can include game themes, software or program instructions todefine and generate a virtual world and/or a virtual view. In anotherembodiment, the memory 106 and/or the disk 108 (FIG. 1) can store someor all of the aforementioned virtual world model and design data. Inanother embodiment, the virtual world model and design data is receivedfrom remote sources, for example, the network 122.

An exemplary virtual world model will now be described with reference toFIGS. 4 and 5. FIG. 4 illustrates an exemplary virtual world data model400 diagram including data flow according to an exemplary embodiment.FIG. 5 is a schematic class diagram 500 of a virtual reality world 502including a group of node classes 504 of the virtual world model of FIG.4 according to an exemplary embodiment. The node classes, objects,properties, references, methods and events (i.e., the one or morecomponents that define the virtual view) discussed with FIGS. 4 and 5are exemplary in nature and are not intended to be limiting. Generally,a virtual world model is a collection of many individual operations andobjects that define the virtual world and one or more virtual views. Thevirtual world model can be defined in various modeling and programminglanguages, for example, virtual reality modeling language (VRML),DirectX, OpenGL, Unity, among others. As illustrated in FIG. 5, thevirtual world 502 can include one or more virtual views 506. The virtualworld 502 and the virtual view 506 can also include one or more virtualobjects 510. The virtual view 506 and the virtual object 510 are definedby node classes, and in the example of FIG. 5, the group of node classes504. In some embodiments, nodes can be grouped and applied to one ormore virtual views 506 and/or virtual objects 510 in a hierarchystructure. For example, the group of node classes 504 can apply toparent and child nodes of the virtual views 506 and/or virtual objects510 associated with a particular group of nodes (i.e., the group of nodeclasses 504).

In the illustrated embodiment of FIG. 5, the group of node classes 504includes a view node class 512, a world structure node class 514, aneventsIn node class 516 and an eventsOut node class 518. Each node classcan include components that define and/or modify the node classes, forexample, other nodes, properties, fields, methods and/or references. Insome embodiments, the nodes, properties, fields, methods and/orreferences can be predefined based on the virtual world, for example,for a particular theme, game, among others. Additionally, in someembodiments, the nodes, properties, fields, methods and/or referencescan be predefined based on the user, for example, based on userpreferences. Other node classes, properties, fields, methods and/orreferences not included in FIGS. 4 and 5 can be implemented and may bebased on the different modeling and programming languages mentionedabove.

The group of node classes 504 of FIG. 5 will now be discussed in detail.The view node class 512 defines the position and/or the orientation ofthe virtual view 506, the virtual object 510 and/or operationsassociated with the virtual view 506 and/or the virtual object 510. Forexample, in FIG. 5, the transform node can be used to perform geometrictransformations and includes the properties position, rotation andscale. The imageEffect node handles image post processing effects.Exemplary image post processing effects include, depth of field, motionand blur, among others. The behavior node can be used to enable anddisable different behaviors, for example, animation and motion. ThevisualEffects node can be used to define visual effects, for example,line renderers, halo effects, trail renders, among others. The renderingnode can be used to define settings and components for rendering in-gameand user interface elements.

The world structure class node 514 defines the structure and appearanceof the virtual view 506, the virtual object 510 and/or operationsassociated with the virtual view 506 and/or the virtual object 510. Forexample, the geometry node can be used to define shapes. The appearancenode can be used to define texture and material. The terrain node can beused to define aspects of a terrain and landscape. The collision nodedefines which objects in a virtual view are collidable.

The eventsIn class node 516 defines the types and names of events thateach node can receive or generate. For example, the EventManager nodecan define custom events and includes event handlers and listeners thatdetermine event triggers (e.g., determined from user data (e.g., userinput), vehicle data) to initialize specific events. The type fielddefines a type of the event, the node field defines which node(s) theevent applies to and the set_fieldname method can be used to modify avalue of a field during the event. The eventsOut class node 518 managesexecution and routing of the event. The fieldname_changed fieldindicates what field(s) are changed during the event, the type fielddefines the type of the event, the node field can define which node theevent applies to and the route method defines how the event is sent to anode and how the event is received and generated by a node. Again, theclass nodes and components in FIG. 5 are exemplary in nature and otherclass nodes and components can be implemented with the systems andmethods discussed herein. The class nodes and components can beaugmented according to at least one of the vehicle data and the userdata to generate a dynamic virtual world and/or virtual views to a user.Specifically, the vehicle data and the user data can be used toinitialize nodes, set properties and fields and initialize or defineevents.

Referring again to FIG. 4, the model 400 includes, one or morecomponents that define a virtual view. For example, in FIG. 4, the modelincludes a view class node 402, a world structure class node 404, aneventsIn class node 406 and an eventsOut class node 408. The class nodesin FIG. 4 can include similar methods, properties, fields and referencesas the class nodes described with FIG. 5. FIG. 4 also illustratesexemplary data flow to the class nodes, for example, for augmenting theclass nodes. Specifically, the dynamic VR module 306 can use these typesof data to augment specific class nodes. As discussed in FIG. 3, thedata can include vehicle data 318, user data 320 and/or other data 322.The types of data illustrated in FIG. 4 are types of vehicle data 318,user data 320 and/or other data 322 in FIG. 3. Specifically, in FIG. 4,the data includes, but is not limited to, vehicle motion data (includingvehicle dynamics data) 412, user motion data 413, vehicle motionpredictive data 414, navigation data 416, big data 418 and driver actiondata 412.

The dynamic VR module 306 modifies and/or augments one or morecomponents of the virtual world model 400 based on at least one of thevehicle data and the user data. The arrows in FIG. 4 from the differenttypes of data to the VR model components, illustrate exemplary flow ofdata that can be used to augment specific VR model components. Forexample, vehicle motion data 412 (e.g., vehicle dynamics data, vehiclevelocity, direction, acceleration, jerk, vehicle occupant motion data)can be used to augment components of the view class node 402. Usermotion data 413 (e.g., position, orientation, location, input) can beused to augmented components of the view class node 402. Further vehiclemotion predictive data 414 (e.g., pedal positions, auto cruise control)can also be used to augment the view class node 402. In anotherembodiment, the navigation data 416 (e.g., navigation location,directions) can be used to augment the world structure class node 404.The big data 418 (e.g., speed bumps, road conditions, steeringconditions) can also be used to augment the world structure class node404. Further, the big data 418 can be used to augment the eventsIn classnode 406. The driver action data 420 (e.g., fuel efficiency, driverinput, audio) can also be used to augment the eventsIn class node 406.

As discussed above, the view class node 402, the world structure classnode 404, the eventsIn class node 406 and the eventsOut class node 408define the virtual view and can be augmented using at least one ofvehicle data and user data to provide a dynamic virtual view to a user.In particular, the data types in FIG. 4, can be used to augment thecomponents of the virtual world model 400 thereby generating a virtualview that can include one or more virtual events 410. In one embodiment,the dynamic VR model 306 augments one or more properties of the one ormore components of the virtual world model based on the vehicle data andthe user data. For example, the one or more properties of the one ormore components can include those properties of the class nodesillustrated in FIG. 5. In one embodiment, one or more of the propertiescan include a motion property defining a motion of the component. Forexample, the view class node 402 can include a property, for example,transform class node that defines the position, the rotation and or thescale of an object. Based on at least one of the vehicle data and theuser data, the transform class node can be augmented to change theposition, rotation and or scale of the object. As an illustrativeexample, and referring to FIGS. 4 and 5, a VR object 510 can be definedas a ball (i.e., defined by the world structure class node 514, forexample, the geometry class node). The vehicle motion data 412 (e.g.,vehicle dynamics data) can be used to augment a feature of the ball. Forexample, using the view class node 402 and the transform class node, theposition, rotation and/or the scale of the ball can be set based on thevehicle motion data 412. Accordingly, the VR object 510, (i.e., theball) is synchronized with the vehicle motion data 412.

Referring again to FIG. 3, the rendering module 308 renders the virtualview from the dynamic VR module 306 to an output device by controllingthe output device to update display of the virtual view according to thevehicle dynamics data. For example, the dynamic reality module 306renders the virtual view to one or more of the VR devices 316 (i.e., theHMD 206, the portable device 208). In one embodiment, the renderingmodule 308 determines vehicle motion data 412 based on the vehicledynamics data (i.e., the vehicle data 318). The rendering module 308 canalso determine user motion data 412 representing motion of the userrelative to the vehicle based on the user data 320 and the vehicle data318. For example, in one embodiment, the VR devices 316 can includeaccelerometer sensors and/or gyroscope sensors that help determine aposition, a location and/or an orientation of the user in relation tothe vehicle. The vehicle motion data 412 and/or the user motion data 412can be used to augment one or more components of the virtual world model400, thereby controlling the output device to update display of thevirtual view according to the vehicle dynamics data. In anotherembodiment, the rendering module 308 can augment the rendering speed(e.g., the frames per second, frame rate, frame update rate defined andimplemented by the graphics rendering hardware/software) of the VRdevices 316 directly based on the vehicle motion data 412 and/or theuser motion data 412.

In a further embodiment, the rendering module 308 determines atemporal-motion rendering speed based on the vehicle motion data and theuser motion data. The temporal-motion rendering speed is a correlationbetween the vehicle motion data and the user motion data. In anotherembodiment, the temporal-motion rendering speed also considers a timecomponent from the vehicle data. The temporal-motion rendering speed isa post image processing and rendering property (e.g., frames per second,frame rate, frame update rate) that minimizes the difference between thevehicle motion data and the user motion data. The rendering module 308can render the virtual view to the output device by controlling theoutput device to update display of the virtual view based on thetemporal-motion rendering speed. For example, the rendering speed (e.g.,the frames per second, frame rate, frame update rate) implemented by thegraphics hardware and/or software of the VR devices 316 can be augmentedbased on the temporal-motion rendering speed. In another embodiment, thedynamic VR module 306 augments one or more properties of the one or morecomponent of the virtual world model based on the temporal-motionrendering speed. For example, the view class node 402 can include arendering properties and/or properties related to motion (See FIG. 5).In one embodiment, these properties can include frames per second, framerate and/or a frame update rate.

By updating the virtual view according to the vehicle dynamics data inreal-time, the virtual view presented to the user is dynamic andconsiders the vehicle motion and the user motion, thereby simulating thevehicle motion and the user motion in the virtual view in real-time.Said differently, one or more components of the virtual world model aresynchronized based on at least the vehicle data and the user data,including the vehicle dynamics data and the user motion data. Not onlydoes this provide a truly immersive virtual reality environment for theuser, but virtual reality motion sickness can be minimized, because thevirtual view considers the vehicle dynamics and the user motion.

The dynamic virtual reality system illustrated in FIGS. 1-5 describedabove will now be described in operation with reference to a method ofFIG. 6. It will be appreciated that the systems and components discussedabove with references to FIGS. 1-5 can similarly be implemented with themethod of FIG. 6. The method of FIG. 6 includes at block 602, receivingvehicle data from one or more vehicle systems of a vehicle, wherein thevehicle data includes vehicle dynamics data. For example, referring toFIG. 3, the VR data module 302 can receive vehicle data 318. The vehicledata 318 can include vehicle dynamics data. Block 602 also includesreceiving user data from a virtual reality device. For example, the VRdata module 302 can also receive user data 320.

Referring back to FIG. 6, at block 604, the method includes generating avirtual view based on the vehicle data, the user data and a virtualworld model. The virtual world model includes one or more componentsthat define the virtual view. FIG. 4 illustrates an exemplary virtualworld model 400 diagram including data flow according to an exemplaryembodiment. The virtual world model 400 includes one or more componentsthat define the virtual view. For example, the view node class 402, theworld structure node class 404, the events in node class 406 and theevents out node class 408 are exemplary components that define thevirtual view. In particular, these node classes define one or morevirtual views and one or more virtual objects of the virtual world. Thenode classes can contain other nodes, properties, fields, methods andreferences (See FIG. 5).

In one embodiment, generating the virtual view includes augmenting oneor more components of the virtual world model according to at least oneof the vehicle data and the user data. For example, the view node class402, the world structure node class 404, the events in node class 406and the events out node class 408, and/or, nodes, properties, fields,methods and references associated with these nodes, can be augmentedbased on at least one of the vehicle data and the user data. In FIG. 4,the vehicle motion data 412, the user motion data 413, the vehiclemotion prediction data 414, the navigation data 416, the big data 418and the driver action data 420 are exemplary types of data that can beused to augment one or more of the components of the virtual world model400.

In one embodiment, the method includes determining an orientation and alocation of the vehicle based on the vehicle data. For example,navigation data 416 (e.g., from for example the GPS 110) can be used todetermine the orientation and the location of the vehicle. The dynamicVR module 306 and/or the rendering module 308 can determine theorientation and the location of the vehicle based on the vehicle data318. The method can also include determining an orientation and alocation of the user relative to the vehicle based on the user data andthe vehicle data. For example, the dynamic VR module 306 and/or therendering module 308 can determine the orientation and the location ofthe user relative to the vehicle based on the user data 320 and thevehicle data 318. For example, the VR devices (e.g., tracking devices,the HMD 202, the portable device 208) can include sensors (e.g.,accelerometers, gyroscopes, compasses) that provide user data fordetermining the orientation and the location of the user relative to thevehicle.

Augmenting one or more components of the virtual world model can bebased on at least one of the orientation and the location of the vehicleand the orientation and the location of the user. As a non-limitingexample, the dynamic VR module can augment the world structure classnode 404, with the orientation and the location of the vehicle and theorientation and the location of the user in relation to the vehicle toprovide real-time world structures. For example, the terrain class node(See FIG. 5) can be augmented to provide a terrain or landscape in thevirtual view that includes a component based on the orientation and thelocation of the vehicle and the orientation and the location of the userin relation to the vehicle.

In another embodiment, one or more virtual events 410 can be augmentedor created based on at least one of the vehicle data 318 and the userdata 320. As an illustrative example, driver action data 420 can be usedto augment the eventsIn class node 406. Driver action data 420, caninclude for example, driving history, fuel efficiency, interactions withother vehicle systems, gestures, motion relative to the vehicle, amongothers. As a non-limiting illustrative example, a user 312 (i.e., avehicle occupant 202), may roll down a vehicle window (not shown). Thisdriver action is determined based on vehicle data 318 from the vehiclesystems 314 (e.g., power window vehicle system) and user data 320 fromthe VR devices 316 (e.g., tracking devices) indicating that the user 312has rolled down the vehicle window. In response, the dynamic VR module306 can augment, for example, the eventIn class node 406, to trigger anevent in the virtual view associated with the user 312 rolling down thevehicle window. For example, a simulation of the wind from the vehiclewindow can be presented in the virtual view, among others. As anotherillustrative example, the dynamic VR module 306 can define the structureof the simulation of the wind based on a temperature determined from thevehicle data 318. For example, if the temperature is a certain degree,the simulation of the wind in the virtual view may include particles ofsnow. This can be defined by augmenting the eventsIn class node 406 ofthe world structure class node 404. The eventsOut node 408 can thenroute the event to create the virtual event 410.

In a further embodiment, the method includes determining vehicle motiondata based on the vehicle dynamics data. The dynamic VR module 306and/or the rendering module 308 can determine the vehicle motion databased on the vehicle data 318, which includes vehicle dynamics data. Asdiscussed above, vehicle motion data 412 define real-time motion of thevehicle. Further, the vehicle motion data can also include predictivevehicle motion data determined based on the vehicle data 318, the userdata 320 and/or the other data 322. For example, the predictive vehiclemotion data can be based on pedal positions, cruise control, destinationinformation, among others. Further, the method can also includedetermining user motion data representing motion of the user relative tothe vehicle based on the user data and the vehicle data. The user motiondata can be based on the vehicle data and the user data 320 and can bedetermined by the dynamic VR module 306 and/or the rendering module 308.

At block 608 the method includes rendering the virtual view to an outputdevice by controlling the output device to update the display of thevirtual view according to the vehicle dynamics data. For example, thedynamic reality module 306 renders the virtual view to one or more ofthe VR devices 316 (i.e., the HMD 206, the portable device 208). In oneembodiment, the rendering module 308 renders the virtual view to anoutput device by controlling the output device to update the display ofthe virtual view according to the vehicle motion data 412 and the usermotion data 413. This can be accomplished, in one embodiment, bydirectly augmenting the rendering speed (e.g., the frames per second,frame rate, frame update rate) implemented by the graphics hardwareand/or software of the VR devices 316. In another embodiment, shown atblock 606, the dynamic VR module 306 augments one or more components ofthe virtual world model 400 according to the vehicle motion data 412 andthe user motion data 413.

In another embodiment, the method includes determining a temporal-motionrendering speed based on the vehicle motion data and the user motiondata. Rendering the virtual view can include controlling the outputdevice to update display of the virtual view based on thetemporal-motion rendering speed. The temporal-motion rendering speed isa correlation between the vehicle motion data 412 and the user motiondata 413. In another embodiment, the temporal-motion rendering speedalso considers a time component from the vehicle data. Thetemporal-motion rendering speed is a post image processing and renderingproperty (e.g., frames per second) that minimizes the difference betweenthe vehicle motion data 412 and the user motion data 413. The renderingmodule 308 can render the virtual view to the output device bycontrolling the output device to update display of the virtual viewbased on the temporal-motion rendering speed. For example, the renderingspeed (e.g., the frames per second, frame rate, frame update rate)implemented by the graphics hardware and/or software of the VR devices316 can be augmented based on the temporal-motion rendering speed.

In a further embodiment, shown at block 606, the dynamic VR module 306augments one or more properties of the one or more component of thevirtual world model based on the temporal-motion rendering speed. Forexample, the view class node 402 can include a rendering propertiesand/or properties related to motion (See FIG. 5). In one embodiment,these properties can include frames per second, frame rate and/or aframe update rate. In this way, the virtual view presented to the useris dynamic and simulates the vehicle motion in real-time. Saiddifferently, one or more components of the virtual world model aresynchronized based on at least the vehicle data and the user data,including the vehicle dynamics data and the user motion data. Not onlydoes this provide a truly immersive virtual reality environment for theuser, but also virtual reality motion sickness can be minimized, becausethe virtual view considers the vehicle dynamics and the user motion.Further, each virtual view presented to the user can be updated andmaintained based on the vehicle dynamics data. For example, if a virtualview presents user settings (i.e., not in a game play mode), the virtualview is always updated and maintained based on the vehicle dynamicsdata.

In some embodiments, virtual reality objects and/or operations can havepredefined rendering speeds. As an illustrative example, certain objectsor certain virtual worlds may be set to rendering speeds of 25 ms. Thispredefined rendering speed can be adjusted based on the vehicle dynamicsdata as discussed above. Thus, in some embodiments, the temporal-motionrendering speed can be based on a predefined rendering speed, vehiclemotion and user motion. Further, the temporal-motion rendering speed canalso consider other types of data based on the user. For example, asdiscussed above, other data 322 can include health data associated withthe user 312. If for example, the health data indicates motion orsensory issues (e.g., disorientation, vertigo, motion sickness), thetemporal-motion rendering speed can be adjusted based on the health datato minimize the motion or sensor issues.

Illustrative examples of dynamic virtual views generated by the methodsand systems discussed herein will now be described with reference toFIGS. 7A and 7B. FIG. 7A illustrates a virtual view 702 and FIG. 7Billustrates a virtual view 704 from a fishing virtual reality game, theobjects and operations of which are defined by a virtual world model,for example the virtual world model 400 of FIG. 4 and the schematicclass diagram 500 of a virtual reality world of FIG. 5. Specifically,the virtual views 702, 704 are generated based on the vehicle data, theuser data and the virtual world model, the virtual world model includingone or more components that define the virtual views 702, 704. Thevirtual views 702, 704 can be displayed on one or more VR devices 314,for example, the HMD 206 and/or the portable device 208 in the vehicle200.

The virtual view 702 includes one or more virtual reality objects,including a landscape 706, a sun 708, a fish 710 and a water object 712.The water object 712 includes one or more waves 714. Again, the objectsand operations of the virtual view 702 are defined by a virtual worldmodel. One or more components of the virtual world model can beaugmented based according to at least one of the vehicle data and theuser data. In particular, in one embodiment, the virtual world model canbe augmented based on vehicle dynamics data and/or the virtual view 702can be rendered to an output device by controlling the output device toupdate display of the virtual view according to the vehicle dynamicsdata. In this way, the virtual view and objects of the virtual view areinfluenced by the vehicle data and/or the user data and the virtual viewand objects of the virtual view are synchronized with the vehicledynamics data. For example, the dynamic VR module 306 can augment theworld structure class node 404 (e.g., the terrain class node) todynamically generate and update the virtual view 702 with objects basedon a location and an orientation of the vehicle 200 and a location andan orientation of the vehicle occupant 202. As an illustrative example,the vehicle 200 is driving in a mountainous region during the day.Accordingly based on the location and the orientation of the vehicle 200and a time component determine from the vehicle data 318, the appearanceand the terrain of the landscape object 706 in the virtual view 702includes mountains and the sun 708. This is accomplished by augmentingthe world structure class node 404 with the location and the orientationof the vehicle 200 and a time component determine from the vehicle data318.

As another example, the fish object 710 can be generated as a type offish indigenous to the location and the orientation of the vehicle 200and the location and the orientation of the vehicle occupant 202.Further, the position of the fish object 710 can also be generated basedon the location and the orientation of the vehicle 200 and the locationand the orientation of the vehicle occupant 202. For example, the viewclass node 402 defining the position and the orientation of the fishobject 710 can be augmented to present a point of view to the vehicleoccupant 202 based on the location and the orientation of the vehicle200 and the location and the orientation of the vehicle occupant 202.

As a further example, the water object 712 in the virtual view 702 canbe generated based on the vehicle dynamics data. As an illustrativeexample, the vehicle data 318 can indicate a steady speed and yaw rateindicating a straight direction. Further, the user data 320 can indicatestead user motion. Accordingly, in FIG. 7A, the waves 714 appear calmand steady based on the vehicle data 318 and the user data 320. However,if the vehicle 200 suddenly increases in speed, as shown in FIG. 7B, thewater object 724 including one or more waves 720 appear rough. In otherembodiments, the water object 724 including the one or more waves 720can be generated by altering the parallax motion or depth of motion ofthe view class node 402 based on the vehicle dynamics data.

In another embodiment, the yaw rate of the vehicle 200 and motion of thevehicle occupant 202 can be used to augment the view class node 402 ofthe boat object to generate a virtual view with the boat turning ormoving based on the yaw rate of the vehicle 200 and the motion of theuser 202. As is apparent, many variations of the virtual view can begenerated based on the vehicle data 318, the user data 320 and the otherdata 322. In addition, in a situation where the virtual view is not in agame play mode (i.e., the virtual view presents game settings, usersettings, start-up instructions), the virtual view is still generatedand updated according to the vehicle dynamics data. Accordingly, byupdating the virtual view according to the vehicle dynamics data inreal-time, the virtual view presented to the user is dynamic andconsiders the vehicle motion and the user motion, thereby simulating thevehicle motion and the user motion in the virtual view in real-time. Notonly does this provide a truly immersive virtual reality environment forthe user, but virtual reality motion sickness can be minimized, becausethe virtual view considers the vehicle dynamics and the user motion.

The embodiments discussed herein can also be described and implementedin the context of computer-readable storage medium storingcomputer-executable instructions. Computer-readable storage mediaincludes computer storage media and communication media. For example,flash memory drives, digital versatile discs (DVDs), compact discs(CDs), floppy disks, and tape cassettes. Computer-readable storage mediacan include volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, modules or otherdata. Computer-readable storage media excludes transitory tangible mediaand propagated data signals.

It will be appreciated that various implementations of theabove-disclosed and other features and functions, or alternatives orvarieties thereof, may be desirably combined into many other differentsystems or applications. Also that various presently unforeseen orunanticipated alternatives, modifications, variations or improvementstherein may be subsequently made by those skilled in the art which arealso intended to be encompassed by the following claims.

The invention claimed is:
 1. A computer-implemented method for virtualreality rendering views to an output device, comprising: receivingvehicle data from a portable device, the vehicle data including avelocity, an orientation, and a location of a vehicle; receiving userdata from the portable device, the user data including a velocity, anorientation and a location of a user; generating a view based on thevehicle data, the user data and a model, the model including one or morecomponents that define the view; determining a difference between theuser data and the vehicle data indicative of a perspective based onmotion of the vehicle from the vehicle data and motion of the user fromthe user data; and rendering the view to the output device bycontrolling the output device to update display of the view according tothe determined difference.
 2. The computer-implemented method of claim1, wherein the user data includes health data about the user, andwherein the health data is received from one or more health monitoringdevices associated with the user.
 3. The computer-implemented method ofclaim 2, wherein the one or more health monitoring devices areintegrated with the output device.
 4. The computer-implemented method ofclaim 2, wherein rendering the view to the output device includesmodifying the view based on the health data to minimize sensoryprocessing.
 5. The computer-implemented method of claim 2, whereinrendering the view is performed at a predetermined rendering speed, andwherein the computer-implemented method includes adjusting thepredetermined rendering speed based on the health data.
 6. Thecomputer-implemented method of claim 1, wherein the output device islocated remotely from the vehicle.
 7. The computer-implemented method ofclaim 1, wherein the vehicle data includes vehicle environment data, andthe computer-implemented method further comprises modifying the viewbased on the vehicle environment data.
 8. The computer-implementedmethod of claim 1, wherein generating the view is further based on amodel, the model including one or more components that define the view,and wherein the one or more components includes a virtual vehicle.
 9. Acomputing system for rendering views to an output device, comprising: aprocessor operatively connected for computer communication to one ormore vehicle systems of a vehicle and the output device, the processorincluding: a virtual reality data module receiving vehicle dataassociated with one or more of the vehicle systems of the vehicle from aportable device, wherein the vehicle data includes a velocity, anorientation, and a location of the vehicle, the virtual reality datamodule receiving predictive vehicle data including a velocity, anorientation, and a location of the vehicle; and the virtual reality datamodule receiving user data the from the portable device, the user dataincluding a velocity, an orientation and a location of a user, whereinthe user is in possession of the output device; a dynamic virtualreality module generates a view based on the vehicle data, thepredictive vehicle data, the user data, and a model, the model includingone or more components that define the view; and a rendering moduledetermines a difference between the user data and the vehicle dataindicative of a perspective based on motion of the vehicle and motion ofthe user from the user data, and wherein the rendering module rendersthe view to the output device by controlling the output device to updatedisplay of the view according to the determined difference and thepredictive vehicle data.
 10. The computing system of claim 9, whereinthe user data includes health data about the user, and wherein thehealth data is received from one or more health monitoring devicesassociated with the user.
 11. The computing system of claim 10, whereinthe one or more health monitoring devices are integrated with theportable device.
 12. The computing system of claim 10, wherein therendering module modifies the view based on the health data to minimizesensory processing.
 13. The computing system of claim 10, wherein therendering module renders the view at a predetermined rendering speed,and wherein the rendering module adjusts the predetermined renderingspeed based on the health data.
 14. The computing system of claim 9,wherein the output device is located remotely from the vehicle.
 15. Thecomputing system of claim 9, wherein the vehicle data includes vehicleenvironment data, and wherein the rendering module modifies the viewbased on the vehicle environment data.
 16. The computing system of claim9, wherein generating the view is further based on a model, the modelincluding one or more components that define the view, and wherein theone or more components includes a virtual vehicle.
 17. A non-transitorycomputer-readable medium comprising instructions that when executed by aprocessor perform a method for rendering views to an output device,comprising: receiving vehicle data from a portable device, the vehicledata including a velocity, an orientation, and a location of a vehicle;receiving user data from the portable device, the user data including avelocity, an orientation and a location of a user; generating a viewbased on the vehicle data, the user data and a model, the modelincluding one or more components that define the view; determining adifference between the user data and the vehicle data indicative of aperspective based on motion of the vehicle from the vehicle data andmotion of the user from the user data; and rendering the view to theoutput device by controlling the output device to update display of theview according to the determined difference.
 18. The non-transitorycomputer-readable medium of claim 17, wherein the user data includeshealth data about the user.
 19. The non-transitory computer-readablemedium of claim 17, wherein the output device is located remotely fromthe vehicle.
 20. The non-transitory computer-readable medium of claim17, wherein the vehicle data includes vehicle environment data.